Several types of networking systems in use today incorporate multiple packet processors. These systems are referred to herein as “multi-packet processor,” or MPP, networking systems. For example, FIG. 1A depicts a stack-based MPP networking system 100 (also known as a “stacking system” or “stack”) that comprises a number of stackable devices 102(1)-102(4) interconnected via external links. In this example, stackable device 102(1) is the master device of the system (denoted by the “M” designation). Each stackable device 102(1)-102(4) (which may be, e.g., a Layer 2/3 switch) includes a management CPU (not shown) for managing the operation of that device (note that the management CPU of the master device is considered the “master CPU” and is responsible for managing the stacking system as a whole). In addition, each stackable device 102(1)-102(4) includes a packet processor 104(1)-104(4) for performing wire-speed processing of network traffic flowing through the device. Although four stackable devices are shown in FIG. 1A, stacking system 100 is modular in nature. Thus, stackable devices may be added to, or removed from, stacking system 100 as needed, thereby increasing or decreasing the number of packet processors in system 100.
As another example, FIG. 1B depicts a chassis-based MPP networking system 150 (also known as a “chassis system”) that comprises a management module 162 and a number of line cards 152(1)-152(3) interconnected via an internal switch fabric 154. Management module 162 includes a management CPU (not shown) for managing the operation of chassis system 150. In addition, each line card 152(1)-152(3) includes a packet processor 156(1)-156(3) for performing wire-speed processing of network traffic flowing through the line card (via respective data ports 158(1)-158(3)). Like stacking system 100, chassis system 150 is modular in nature. Thus, line cards may added to, or removed from, chassis system 150 as needed, thereby increasing or decreasing the number of packet processors in system 150.
In an MPP networking system such as system 100 or 150, some hardware capacities scale with the number of packet processors. For instance, the total port capacity of chassis system 150 is equal to the number of ports supported by each line card/packet processor, multiplied by the number of line cards/packet processors. If additional line cards/packet processors are added, the total port capacity of the system will increase accordingly.
On the other hand, other hardware capacities of an MPP networking system do not scale with the number of packet processors. For instance, consider hardware MAC tables, which are used for Layer 2 switching and are commonly implemented using a hardware (e.g., DRAM or SRAM-based) hash table within each packet processor. Examples of such hardware hash tables are shown via reference numerals 106(1)-106(4) and 160(1)-160(3) in FIGS. 1A and 1B respectively. In a typical Layer 2 switching flow, when a packet processor of an MPP networking system receives a data packet with a destination MAC address that is “unknown” (i.e., is not in the packet processor's hardware MAC table), the packet processor floods the packet throughout the packet's VLAN. The packet processor also checks whether the packet's source MAC address is unknown and, if so, sends a message to the system's master/management CPU to learn the source MAC address. Upon processing the message, the master/management CPU installs the source MAC address to the hardware MAC table of every packet processor in the system (thereby ensuring that the MAC address is present in hardware when a reply to the original data packet is received). Since this flow requires each hardware MAC table to maintain the same set of learned MAC addresses, adding additional packet processors (and thus, additional hardware MAC tables) will not increase the system's hardware MAC table capacity; instead, that capacity will be limited by the size of the smallest hardware MAC table in the system.
Generally speaking, the need to duplicate learned MAC addresses across hardware MAC tables is not an issue if the hardware MAC tables are the same size—in this case, the applications running on the MPP networking system can be programmed to account for the common size limit shared by all of the hardware MAC tables, and thus can manage the tables in a uniform manner. However, this requirement can cause problems if the hardware MAC tables have varying sizes. For example, in stacking system 100 of FIG. 1A, assume that hardware hash tables 106(1)-106(4) of packet processors 104(1)-104(4) are used as hardware MAC tables. Further assume that hardware hash tables 106(1)-106(3) each support 32K entries, while hardware hash table 106(4) only supports 16K entries. In this scenario, if the CPU of master device 102(1) attempts to install a MAC address entry to every hardware hash table in the system, the installation may succeed with respect to tables 106(1)-106(3) (due to their larger size), but may fail with respect to table 106(4) (due to its smaller size). This inconsistency can be very difficult to manage in software, particularly since the stackable devices of stacking system 100 can be added or removed at will. In addition, if left unhandled, the failed MAC installation at packet processor 104(4) can lead to several negative consequences, ranging from increased flooding (if the uninstalled MAC address is a destination in a data flow) to the inability to learn proper network protocol states (if the uninstalled MAC address is a protocol MAC address).
One workaround for the foregoing problem is to constrain the effective hardware MAC table size of each packet processor in an MPP networking system to the smallest hardware MAC table size. For instance, in the example above, stacking system 100 can be configured to use only 16K entries per table, even though hardware hash tables 106(1)-106(3) actually support 32K entries. While this workaround avoids situations where a MAC address entry cannot be consistently installed to all hardware MAC tables, it clearly does not make optimal use of the hardware table capacity of each packet processor.
Another workaround is to implement an alternative MAC learning approach known as “flow-based MAC learning.” With flow-based MAC learning, a data packet with an unknown destination MAC address is trapped to the system's master/management CPU. The master/management CPU then installs the source (or destination) MAC address of the packet, if needed, solely in the hardware MAC table of the ingress packet processor, without duplicating the address to other packet processors in the system. This approach is efficient in terms of hardware table usage since MAC address entries are effectively installed “on demand” at a given packet processor in response to a received traffic flow. Unfortunately, flow-based MAC learning is also very CPU intensive because all packets that fail hardware matching must be processed by the master/management CPU.
The high CPU cost of flow-based MAC learning is particularly problematic in a stacking system like system 100 of FIG. 1A, where a master CPU residing on the system's master device (e.g., master device 102(1)) must generally transmit internal packets to other remote devices in the system in order to install MAC address entries to the hardware MAC tables of those remote devices. These internal packets may be lost, or may be queued for an extended period of time at the remote devices if the CPUs of those devices are busy. During this period, all packets having the yet-to-be-installed MAC address as a destination address will continue to be trapped to the master CPU for processing, thereby further increasing the CPU load of the system.